Augmented reality overlays computer-generated information with a user's physical environment through video integration or a transparent display. Augmented reality merges computer-generated, synthetic information with a user's view of his or her surroundings. For the past two decades, researchers have demonstrated the promise of augmented reality (AR) to allow society to reach new levels of capability and efficiency in such diverse areas as medicine [22], manufacturing [5], maintenance [18], navigation [17], and telepresence [51]. Although to date this technology has been primarily confined to the lab, new advances in mobile processors, sensors, and displays offer the potential of mainstream use of AR. In particular, optical see-through head mounted displays (HMDs), which offer augmented overlays over one's natural vision, are beginning to become practical. The bulky designs and narrow fields of view that failed in the past are being replaced with high-performance devices in comfortable and compact form factors that approach ordinary eyeglasses.
However, the usefulness of conventional augmented reality systems is hindered by the deleterious effects of system latency, which causes an image mismatch between the real and synthetic imagery. This is known as the augmented reality registration problem and has many causes including system latency, tracker error, calibration error, and optical distortion. Studies have shown that the majority of the error is caused by the total system latency, introducing more error than all other issues combined. Non-uniform latency, or jitter, is more noticeable than a constant latency. Latency in conventional augmented reality systems has been observed to incur 1 millimeter of misalignment for every 1 millisecond of latency.
In most cases, augmented imagery is merged via video or optical integration on either a handheld or head-worn display that is constantly moving with a human user. System latency introduces misalignment of the overlain imagery during display motion, as the pose estimate used to generate the image is no longer valid at the time the image is displayed.
As used herein, the term “pose” refers to the apparent position (e.g., the X, Y, and Z coordinates) and orientation (e.g., the azimuth and inclination or the pitch, roll, and yaw) of the viewer, which determines what portion of the real world the viewer sees at a given instant.
Augmented reality has seen a recent resurgence on handheld mobile devices. However, these applications thrive on their ability to mask the detrimental effects of latency. These systems hide the delay by simultaneously delaying the user's view of the real world to synchronize with the overlaid imagery. Delaying the real world is only possible with video see-through image integration where a camera captures the real world and displays it to the user along with synthetic information. This technique does not apply to optical see-through designs since they merge the computer-generated imagery (and its inevitable latency) with a user's natural view of his or her surroundings through transparency (which cannot be delayed). Delaying the real world is a viable option for handheld displays since they use video see-through, typically capturing the real world with a camera on the back of the unit. The method's delay is only viable on a handheld display because the user may not notice the delay induced difference between the image on the display and the underlying view. The delays from this method are much more noticeable on a head-worn display, often inducing simulator sickness.
Optical see-through displays offer a direct and undegraded view of the real environment that is suitable for extended use throughout one's daily activities. However, this desirable attribute comes at a cost; unlike video see-through displays, which allow synchronization of real and virtual objects in software through a combined video signal, optical see-through designs must rely on very low display latency to keep the virtual and real aligned [65]. The latency of today's AR systems, even those optimized for low latency, extends beyond a simple annoyance or distraction and renders many optical see-through applications unusable.
Unfortunately, latency occurs throughout all components of an AR system and thus is not an easy problem to fix [37]. Tracking cameras process data in whole frame intervals and apply processing to ameliorate the effects of rolling shutters. Modern graphics hardware uses deep pipelines to achieve high performance at the expense of latency, and scanout uses one or more frame buffers to prevent image tearing. Displays provide onboard image scaling and enhancement which insert additional delays. The sum of these latencies typically numbers several frames.
As computing is becoming increasingly mobile, there is also an increasing desire for mobile augmented reality. Many sought augmented reality applications are only useful if done on either a hand-held or head-worn device as they provide a custom, individualized augmentation for each user in a shared space. A mobile platform imposes severe constraints on processing, memory, and component size for all activities: tracking, rendering, and display.
There are disadvantages associated with conventional augmented reality systems. A typical augmented reality system uses readily available peripherals and renderers designed for overall performance. Cameras are typically designed to be general purpose and useful in numerous applications. The vast majority of current camera technology is not well suited for onboard pose estimation for head-mounted displays. For augmented reality applications a camera-based head tracker requires a high-frame rate to ensure frequent updates. Unfortunately, high-frame rate cameras are expensive and power hungry, and may result in a reduced signal to noise ratio. The same price and power increases are true of displays. Typical displays and some image sensor controllers use buffers, meaning that data isn't sent until an entire frame is ready. Because the data is ultimately read sequentially, the data towards the end of each frame arrives later than when it was generated resulting in end to end latency.
Accordingly, in light of these disadvantages associated with conventional augmented reality systems, there exists a need for systems and methods for low-latency stabilization for head-worn displays.